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Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 
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earned patent term adjustment. See 37 CFR 1.704(b). 
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1)^ Responsive to communication(s) filed on 28 April 2003 . 
2a)S This action is FINAL. 2b)n This action is non-final. 

3) n Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex paAte Quayle, 1935 CD. 11, 453 O.G. 213. 
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4) S Claim(s) 1-20 is/are pending in the application. 
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5) n Claim(s) is/are allowed. 

6) 13 Claim(s) 1-4.6 and 14-20 is/are rejected. 

7) ^ Claim(s) Sand 7-13 is/are objected to. 

8) n Claim(s) are subject to restriction and/or election requirement. 

Application Papers 
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DETAILED ACTION 

Remarks 

L In response to communications filed on 28-April-2003, claims 1-20 are pending in the 
application. 

Claim Rejections - 35 USC §103 
2. The following is a quotation of 35 U.S. C. 103(a) which forms the basis for all obviousness 
rejections set forth in this Office action: 



(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentabihty shall not be negatived by the 
manner in which the invention was made. 

3. Claims 1-3 and 6 are rejected under 35 U.S.C. 103(a) as being unpatentable over Ma et al 
(U.S. patent No. 6,347,3 13) in view of Liddv et al (U.S. patent No. 6,304,864.) 
As to claim 1, Ma et al teaches a multimedia data structure reflecting change of a user 

relevance feedback (see column 1, lines 5-10) for determining weights of image features used for 

an image search, comprising: 

(a) information describing at least one feature of a certain image (see column 3, Hnes 18-23); 

and 
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(b) recent user feedback information (see column 3, lines 37-56) based on user relevance 
feedback (see column 6, lines 45-50.) 

Ma et al does not teach: 

(c) whole feedback information based on the user relevance feedback. 

Liddy et al teaches a system for retrieving multimedia information (see Abstract), in which 
she teaches whole feedback information based on the user relevance feedback (see column 12, 
lines 9-24, and see column 13, lines 26-38, where "whole feedback" is read on "relevance 
feedback" on a "periodic time interval".) 

Therefore, it would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have modified Ma et al to include whole feedback information based 
on the user relevance feedback. 

It would have been obvious to a person having ordinary skill in the art at the time the 
invention was made to have modified Ma et al by the teaching of Liddy et al because having 
whole feedback information based on the user relevance feedback, would enable the system to 
define periodic intervals automatically, or as set by the user, to allow accumulation of all 
relevance feedback on a particular object (image) to be captured, in order to categorize the 
objects (image) based on whole user relevance feedback. 

As to claim 2, Ma et al as modified teaches wherein the recent user feedback information is 
determined for a predetermined time period (see Liddy et al. column 12, hnes 13-14, where 
"predetermined time period" is read on "interval may be 15 minutes") or by a predetermined 
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frequency (see Liddv et al column 12, lines 21-22, where "predetermined frequency" is read on 
"X number of documents".) 

As to claim 3, Ma et al as modified teaches wherein the recent user feedback information is a 
weight value learned by the user relevance feedback or a similar image information (see Ma et 
al, column 6, lines 49-50, and see column 8, lines 4-10), and the whole feedback information is 
represented by a weight value learned by previous feedback (see Liddv et al , column 13, lines 
30-33.) 

As to claim 6, Ma et al as modified teaches the method comprising: 

representing the recent user feedback information by a similar image list (see Liddv et al , 

column 8, lines 60-64); and 

reflecting a recent user feedback pattern by the similar image list (see Liddv et aL column 3, 

line 64 trough column 4, line 9), using a queue algorithm (see Liddv et al , column 10, lines 34- 

38.) 

4. Claims 4 and 14-20 are rejected under 35 U.S.C. 103(a) as being unpatentable over Ma et al 
(U.S. patent No. 6,347,3 13) m view of Liddv et al (U.S. patent No. 6,304,864), as applied to 
claims 1-3 and 6 above, and further in view of Cohen (U.S. Patent No. 6,067,539.) 

As to claim 4, Ma et al as modified does not teach the data structure further comprising 
recent user feedback reliability information representing how reUable the recent user 
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feedback information is, and whole feedback reliability information representing how 
reliable the whole feedback information is. 

Cohen teaches an intelligent information retrieval system (see Abstract), in which he 
teaches: recent user feedback reliability information representing how reUable the recent 
user feedback information is (see column 7, lines 51-56), and whole feedback rehability 
information representing how reUable the whole feedback information is (see column 2, lines 
45-64, where "whole feedback" is read on "updating the score with scores received on 
previous message".) 

Therefore, it would have been obvious to a person having ordinary skill in the art at the 
time the invention was made to have modified Ma et al as modified to include recent user 
feedback reliability information representing how rehable the recent user feedback 
information is, and whole feedback rehability information representing how reliable the 
whole feedback information is. 

It would have been obvious to a person having ordinary skill in the art at the time the 
invention was made to have modified Ma et al as modified, by the teaching of Cohen , 
because recent user feedback reliability information representing how reliable the recent user 
feedback information is, and whole feedback reliability information representing how 
rehable the whole feedback information is, would increase the efficiency and accuracy of the 
entered feedback and allow data (images) with the most relevant/reliable user feedback to 
receive a higher rank/weight for presentation to the user than data (images) with a lower 
reliability score. 
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As to claim 14, Ma et al teaches a method of determining weights of image features used for 
an image search based on user relevance feedback (see Abstract), comprising: 

(a) providing a multimedia data structure (see column 1, lines 5-10) including information 
describing the features of a certain image (see column 3, lines 18-23), and recent user feedback 
information for the image (see column 3, lines 37-56, and see column 6, lines 45-50); 

(b) updating the recent user feedback information (see column 2, lines 28-33); 

(c) determining weights of the image features in proportion to the reliabilities of the recent 
feedback information (see column 6, lines 1 1-18.) 

Ma et al does not teach: whole feedback information for the image; and determining weights 
of whole feedback information, or both the recent feedback information and the whole feedback 
information. 

Liddy et al teaches a system for retrieving multimedia information (see Abstract), in which 
/ she teaches: whole feedback information for the image (see column 12, lines 9-24, and see 
column 13, lines 26-38, where "whole feedback" is read on "relevance feedback" on a "periodic 
time interval"); and determining weights of whole feedback information, or both the recent 
feedback information and the whole feedback information (see column 13, lines 15-18.) 

Therefore, it would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have modified Ma et al to include whole feedback information for the 
image; and determining weights of whole feedback information, or both the recent feedback 
information and the whole feedback information. 

It would have been obvious to a person having ordinary skill in the art at the time the 
invention was made to have modified Ma et al by the teachings of Liddv et al because obtaining 
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whole feedback information for the image; and determining weights of whole feedback 
information, or both the recent feedback information and the whole feedback information, would 
enable the system to define periodic intervals automatically, or as set by the user, to allow 
accumulation of all relevance feedback on a particular object (image) to be captured, in order to 
categorize the objects (image) based on whole user relevance feedback and based on the weights 
(ranking) of relevance of the entered feedback. 

Ma et al as modified still does not teach reliability information corresponding to the 
recent user feedback information and whole feedback information; and updating whole 
feedback information and their reliabilities by learning them in response to the user relevance 
feedback. 

Cohen teaches an intelligent information retrieval system (see Abstract), in which he 
teaches reliability information corresponding to the recent user feedback information (see 
column 7, lines 51-56), and whole feedback information (see column 2, lines 45-64, where 
"whole feedback" is read on "updating the score v^th scores received on previous message"); 
and updating whole feedback information and their reliabilities by learning them in response 
to the user relevance feedback (see column 7, lines 36-38.) 

Therefore, it would have been obvious to a person having ordinary skill in the art at the 
time the invention was made to have modified Ma et al as modified to include reliability 
information corresponding to the recent user feedback information and whole feedback 
information; and updating whole feedback information and their reliabilities by learning 
them in response to the user relevance feedback. 
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It would have been obvious to a person having ordinary skill in the art at the time the 
invention was made to have modified Ma et al as modified, by the teaching of Cohen , 
because reliability information corresponding to the recent user feedback information and 
whole feedback information; and updating whole feedback information and their reUabilities 
by learning them in response to the user relevance feedback, would increase the efficiency 
and accuracy of the entered feedback and allow data (images) with the most relevant/rehable 
user feedback to receive a higher rank/weight for presentation to the user than data (images) 
with a lower reliability score. 

As to claim 15, Ma et al as modified teaches wherein the recent user feedback 
information is represented by a weight value learned by the user relevance feedback or a 
similar image information (see Ma et al , column 6, lines 49-50, and see column 8, lines 4- 
10), and the whole feedback information is represented by a weight value learned by 
previous feedback (see Liddv et al column 13, lines 30-33.) 

As to claim 16, Ma et al as modified teaches wherein the reliability of the recent user 
feedback information (see Cohen , column 7, Unes 51-56) is determined in proportion to a 
consistency of a recently used pattern or feedback (see Liddy et al column 3, line 64 trough 
column 4, line 9.) 
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As to claim 17, Ma et al as modified teaches wherein the reliability of the whole feedback 
information is determined in proportion to the number of feedback concerned in learning (see 
Cohen , column 9, lines 32-51.) 

As to claim 18, Ma et al as modified teaches wherein the reliability of the whole feedback 
information is responsive to recorded usage records wherein the recorded user usage records 
provide feedback to the reliability of the whole feedback information without user interaction 
(see Cohen , column 4, lines 31-39.) 

As to claim 19, the appUcant is directed to the remarks and discussions made in claim 14 
above. 

As to claim 20, Ma et al as modified teaches wherein the reliability information indicates 
reUability of both the user feedback information (see Cohen, column 7, lines 51-56) and the 
whole feedback information (see Cohen , column 2, lines 45-64, where "whole feedback" is 
read on "updating the score with scores received on previous message.) 

Allowable Subject Matter 
5. Claims 5 and 7-13 are objected to as being dependent upon a rejected base claim, but would 
be allowable if rewritten in independent form including all of the hmitations of the base 
claim and any intervening claims. 
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6. The following is a statement of reasons for the indication of allowable subject matter: 

The prior art of record, Ma et al (U.S. Patent No. 6,347,3 13), Liddv et al (U.S. Patent No. 
6,304,864), and Cohen (U.S. Patent No. 6,067,539) do not disclose, teach, or suggest the claimed 

limitations of (in combination with all other features in the claims): 

wherein the recent user feedback reliability information is expressed by 



where, N is the number of feedback, m is the number of images in the similar image Ust, and 
n(i) is the number of feedback given to the i-th image, as claimed in claim 5. 

The prior art of record, Ma et al (U.S. Patent No. 6,347,3 13), Liddv et al (U.S. Patent No. 
6,304,864), and Cohen (U.S. Patent No. 6,067,539) do not disclose, teach, or suggest the claimed 
limitations of (in combination with all other features in the claims): 

(b)if it is checked that the corresponding object does not exist in the current queue, inputting 
the corresponding object to an uppermost space of a queue entrance, setting the number of 
feedback of the corresponding object to "1", and shifting objects existing in the queue to lower 
positions by one space; 
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(c) if it is checked that the corresponding object exists in the current queue, increasing the 
number of feedback of the corresponding object, and shifting the objects existing in the queue to 
upper positions by "N"; and 

(d) if any object is shifted to the lower position over a size of the queue at the respective 
steps, deleting the corresponding object from the queue, as claimed in claim 7. 

Claims 8-10 are objected to as being dependent upon the objected to dependent claim 7. 

The prior art of record, Ma et al (U.S. Patent No. 6,347,3 13), Liddv et al (U.S. Patent No. 
6,304,864), and Cohen (U.S. Patent No. 6,067,539) do not disclose, teach, or suggest the claimed 
limitations of (in combination with all other features in the claims): 

wherein the recent user feedback information is represented as a similar image list, and the 
similar image list has an image list structure composed of a similar image identification, a score 
reflecting the current feedback, and a waiting duration representing a time period between the 
final feedback time point and the present time point, as claimed in claim 11. 

Claims 12-13 are objected to as being dependent upon the objected to dependent claim 11. 

Response to Arguments 
7. Applicant's arguments filed on 28-April-2003 with respect to the cited references have been 
fiiUy considered but they are not found to be persuasive: 
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In response to applicants' arguments that "the cited references must teach or suggest all 
the features in claim 1", and that "the Ma patent does not teach or suggest a data structure 
used for determining weights of image features of an image search which includes all user 
feedback information input up to a current point in time", the arguments have been fully 
considered but are not found to be persuasive, because Ma et al teaches a multimedia data 
structure (see column 1, lines7-9, where "multimedia data structure" is read on "content - 
based multimedia retrieval") reflecting change of a user relevance feedback (see column 1, 
lines 9-10) for determining weights of image features used for an image search (see colunin 
6, lines 44-60, and see column 2, hnes 55-67), which includes recent user feedback 
information (see column 3, lines 37-56.) The "whole feedback" information is taught by the 
secondary reference, Liddy et al (see column 12, Unes 9-24, and see column 13, lines 26-38, 
where "whole feedback" is read on "feedback on a periodic time interval".) 

In response to appUcants arguments that "there must have been some teaching or 
suggestion in existence at the time the claimed invention was made that would have led one 
of ordinary skill in the art to combine the references in an attempt to form the invention", and 
that "none of the references of record provide such teaching or suggestion", the arguments 
have been fully considered but are not found to be persuasive, because the examiner 
recognizes that obviousness can only be established by combining or modifying the teachings 
of the prior art to produce the claimed invention where there is some teaching, suggestion, or 
motivation to do so found either in the references themselves or in the knowledge generally 
available to one of ordinary skill in the art. See//? re Fine, 837 F.2d 1071, 5 USPQ2d 1596 
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(Fed. Cir. 1988) and In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992). In this 
case, the examiner is estabUshing obviousness in the knowledge generally available to one of 
ordinary skill in the art, to modify the invention of Ma et al with the teaching of Liddv et al 
as explained in the previous office action. 

In response to applicants arguments that "the Liddy patent does not teach or suggest that 
the neural network is based on whole feedback information , e.g., all user feedback 
information input up to a current point in time", the arguments have been fully considered 
but are not found to be persuasive, because Liddy et al teaches obtaining feedback from the 
users over a "time period" (see column 12, lines 9-24, and see column 13, lines 26-38, where 
"evolution time" is taught by Liddy et al to be "a clock time set by the user via the GUI", 
which indicates that the clock could be set on any desired time period and/or interval.) 

In response to appUcants arguments that "the Cohen patent does not disclose including 
both recent user feedback and whole feedback information in the same data structure ", and 
that "Cohen does not provide a teaching or suggestion for motivating one of ordinary skill in 
the art to modify a data structure including recent user feedback information to also include 
whole feedback information", the arguments have been fully considered but are not found to 
be persuasive, because the combination of Ma et al and Liddy et al patents estabUshed the 
above teachings, as explained in claim 1 above. Cohen is cited by the examiner for the 
teachings of "feedback rehability" recitation, as explained in the previous office action. 
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Conclusion 



8. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy 
as set forth in 37 CFR LI 36(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until 
after the end of the THREE-MONTH shortened statutory period, then the shortened statutory 
period will expire on the date the advisory action is mailed, and any extension fee pursuant to 
37 CFR 1. 136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 

9. Any inquiries concerning this communication or earlier communications from the examiner 
should be directed to Tony Mahmoudi whose telephone number is (703) 305-4887. The 
examiner can normally be reached on Mondays-Fridays from 08:00 am to 04:30 pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Dov Popovici, can be reached at (703) 305-3830. 
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